Recording a Command Stream with a Rich Encoding Format for Capture and Playback of Graphics Content

ABSTRACT

Analyzing an application executing on a target device. An application may be executed on a target device. Low cost measurement may be gathered regarding the application executing on the target device. In response to a trigger, high cost measurement data may be gathered regarding the application executing on the target device. The high cost measurement data may include graphics commands provided by the application. The graphics commands and related information may be stored and provided to a host. The host may modify the graphics commands to perform experiments to determine performance issues of the application executing on the target device. The host may determine whether the performance is limited by the CPU or the GPU and may determine specific operations that are causing performance issues. The host may provide suggestions for overcoming the performance issues.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/834,344, filed on Aug. 24, 2015, which is a continuation of U.S.patent application Ser. No. 14/139,467, filed on Dec. 23, 2013, which isa continuation of U.S. patent application Ser. No. 12/896,097, filed onOct. 1, 2010, the entire contents of each of which are incorporatedherein by reference.

FIELD OF THE INVENTION

The present invention relates to the field of computer graphics, andmore particularly to performance analysis of graphics applications.

DESCRIPTION OF THE RELATED ART

Many modern devices include high end graphical processing systems forpresenting graphics on a display. Due to their complexity, developmentof graphical software applications for such devices is often difficult.For example, development of software which interacts with the graphicalprocessing system often results in sub-optimal graphical systemperformance and resource utilization. In order to assist developers increating graphical software applications, improved tools are desired fordetection of performance bottlenecks and opportunities for performanceoptimization in graphical application development.

SUMMARY OF THE INVENTION

Various embodiments are presented of a system and method for analyzingapplications executing on a target device. In particular, embodimentsare presented for analyzing the performance of graphics applications(applications which generate graphics for display).

The target device and a host device may be paired together. The hostdevice may execute a development application to perform testing of anapplication on the target device. The host device may deploy theapplication to the target device and begin execution of the applicationon the target device. The host device may also deploy other programs onthe target device in order to analyze the execution of the applicationon the target device. For example, the host device may deploy one ormore measurement or monitoring applications which may perform monitoringof the execution of the application while it executes on the targetdevice.

For example, the measurement programs may initially perform low costmeasurement of the application while it executes on the target device.The low cost measurement of the application may not substantially impactperformance of the application execution. The low cost measurement mayprovide execution information, such as CPU load (e.g., related tographics processing or not), GPU load, frame rate, etc.

The target device may include a graphics system which includes both acentral processing unit (CPU) and a graphics processing unit (GPU).During execution of the graphics application on the CPU of the targetdevice, the measurement software may measure execution load of the CPUand the GPU of the device and determine whether the CPU is limitinggraphics performance of the graphics application. If the CPU is limitinggraphics performance of the graphics application, the measurementsoftware may provide an indication to the host computer. The applicationdeveloper can then use this information to modify the application.

During operation of the low cost measurement, the measurement programmay monitor for various conditions (or “triggers”) which indicate aperformance issue that merits more detailed monitoring. In response to atrigger, high cost measurement may be initiated (e.g., by themeasurement application). The high cost measurement involves a moredetailed analysis of system performance, which is hence more “costly” ormore “intrusive”. The high cost measurement may be provided viaautomatic processes (e.g., a detected drop in frame rate above athreshold amount, increase in GPU load a threshold amount, etc.) ormanually (e.g., a user providing input to invoke the trigger).

During high cost measurement, graphics commands provided by theapplication may be captured and recorded. For example, graphics commandsprovided from the application to a graphics framework (also executing onthe target device) may be intercepted by the measurement application.The measurement application may then store these commands and may alsoderive additional information regarding the commands or the state of thegraphics system of the target device. The additional information maycomprise data such as: a timestamp indicating a time when the firstgraphics command was received or executed; a duration of time forexecution of a first graphics command; state information indicating anexecution state of the application; a current graphics framework errorat the time of receiving a respective graphics command; a flagindicating that the first graphics command should not be executed onlater playback of the plurality of graphics commands; and/or a graphicscommand that is not provided by the graphics application, among numerousothers. The additional information may be used in conjunction with therecorded command stream for later playback of the commands or foranalysis, as desired. The measurement application may also store relatedgraphics information (e.g., textures referenced by the graphicscommands).

After completion of the high cost measurement, a command stream as wellas additional information may be stored by the host device. In oneembodiment, the command stream may be aggregated by the host device inresponse to provision of the commands, additional data, and associatedgraphics data by the target device.

The host device may generate one or more modifications to the commandstream in order to determine performance issues or bottlenecks of theapplication that were present during the high cost measurement. Forexample, the modifications may disable or simplify various portions ofthe graphics pipeline or individual operations in order to determine acause of a performance issue. In some embodiments, the modifications maycomprise a modification to one or more of a shader, resource, orgraphics state during execution of the modified command stream.

The modifications may be used to generate one or more modified commandstreams (e.g., by the host or the target device), which may then beexecuted by the target device. The modified command stream(s) may beexecuted by a player application that may be deployed on the targetdevice. The player application may be configured to execute the variousmodified command streams. The target device may monitor execution of thevarious modified command streams in order to gather performanceinformation of that respective modified command stream.

Accordingly, the various modifications to the command stream may beexecuted and performance data may be gathered for each execution of themodified command stream. The performance data may be analyzed toidentify performance issues of the application executing on the targetdevice. Stated another way, the host may “try out” differentmodifications to the recorded command stream to attempt to isolaterespective portions of the application software that are causingbottlenecks or performance issues when run on the target device. Oncethe performance issues are identified, one or more suggestions may beprovided (e.g., to the developer using the development program) toovercome or address these performance issues.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description of the preferred embodiment is consideredin conjunction with the following drawings, in which:

FIGS. 1A-1E illustrate various systems for implementing variousembodiments of the present invention;

FIGS. 2A and 2B are block diagrams of a system including a CPU and aGPU, according to some embodiments;

FIG. 3 is a block diagram of software executing on a host device and atarget device, according to one embodiment;

FIGS. 4A-B are exemplary block diagrams illustrating embodiments of agraphics pipeline;

FIG. 5 is a flowchart diagram illustrating one embodiment of a methodfor analyzing execution of an application on a target device;

FIG. 6 is a flowchart diagram illustrating one embodiment of a methodfor switching from low cost monitoring of execution of an application tohigher cost monitoring in response to a trigger;

FIG. 7 is a flowchart diagram illustrating one embodiment of a methodfor capturing a command stream from an application executing on a targetdevice;

FIG. 8 is a flowchart diagram illustrating one embodiment of a methodfor determining performance issues of an application executing on atarget device;

FIG. 9 is a flowchart diagram illustrating one embodiment of a methodfor determining whether the CPU is limiting graphics performance of anapplication executing on a target device; and

FIGS. 10A-10E are exemplary GUIs for an analysis program, according toembodiments described herein.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and are herein described in detail. It should beunderstood, however, that the drawings and detailed description theretoare not intended to limit the invention to the particular formdisclosed, but on the contrary, the intention is to cover allmodifications, equivalents and alternatives falling within the spiritand scope of the present invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE EMBODIMENTS Terms

The following is a glossary of terms used in the present application:

Memory Medium—Any of various types of memory devices or storage devices.The term “memory medium” is intended to include an installation medium,e.g., a CD-ROM, floppy disks 104, or tape device; a computer systemmemory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM,Rambus RAM, etc.; a non-volatile memory such as a Flash, magnetic media,e.g., a hard drive, or optical storage; registers, or other similartypes of memory elements, etc. The memory medium may include other typesof memory as well or combinations thereof. In addition, the memorymedium may be located in a first computer in which the programs areexecuted, or may be located in a second different computer whichconnects to the first computer over a network, such as the Internet. Inthe latter instance, the second computer may provide programinstructions to the first computer for execution. The term “memorymedium” may include two or more memory mediums which may reside indifferent locations, e.g., in different computers that are connectedover a network. The memory medium may store program instructions (e.g.,embodied as computer programs) that may be executed by one or moreprocessors.

Carrier Medium—a memory medium as described above, as well as a physicaltransmission medium, such as a bus, network, and/or other physicaltransmission medium that conveys signals such as electrical,electromagnetic, or digital signals.

Computer System—any of various types of computing or processing systems,including a personal computer system (PC), mainframe computer system,workstation, network appliance, Internet appliance, personal digitalassistant (PDA), personal communication device, smart phone, televisionsystem, grid computing system, or other device or combinations ofdevices. In general, the term “computer system” can be broadly definedto encompass any device (or combination of devices) having at least oneprocessor that executes instructions from a memory medium.

Automatically—refers to an action or operation performed by a computersystem (e.g., software executed by the computer system) or device (e.g.,circuitry, programmable hardware elements, ASICs, etc.), without userinput directly specifying or performing the action or operation. Thusthe term “automatically” is in contrast to an operation being manuallyperformed or specified by the user, where the user provides input todirectly perform the operation. An automatic procedure may be initiatedby input provided by the user, but the subsequent actions that areperformed “automatically” are not specified by the user, i.e., are notperformed “manually”, where the user specifies each action to perform.For example, a user filling out an electronic form by selecting eachfield and providing input specifying information (e.g., by typinginformation, selecting check boxes, radio selections, etc.) is fillingout the form manually, even though the computer system must update theform in response to the user actions. The form may be automaticallyfilled out by the computer system where the computer system (e.g.,software executing on the computer system) analyzes the fields of theform and fills in the form without any user input specifying the answersto the fields. As indicated above, the user may invoke the automaticfilling of the form, but is not involved in the actual filling of theform (e.g., the user is not manually specifying answers to fields butrather they are being automatically completed). The presentspecification provides various examples of operations beingautomatically performed in response to actions the user has taken.

FIGS. 1A-1E—Exemplary Systems

FIG. 1A illustrates a computer system 100 (host) that is coupled to atarget device 150. The host computer system 100 may be any of variouscomputer systems. The target device 150 may also be any of variouscomputer systems. In some embodiments, the target device 150 may be aportable or mobile device, such as a mobile phone, PDA, audio/videoplayer, etc. In embodiments described herein, the computer system 100may be configured to act as a host device, which may manage execution ofan application (e.g., a graphics application) on the target device 150,e.g., for application development and/or performance analysis, asdescribed herein.

As shown in FIG. 1A, the computer system 100 may include a displaydevice configured to display a graphical user interface (GUI), e.g., ofa control or development application executing on the computer system100. The graphical user interface may include any type of graphical userinterface, e.g., depending on the computing platform. The computersystem 100 may include at least one memory medium on which one or morecomputer programs or software components according to one embodiment ofthe present invention may be stored. For example, the memory medium maystore the control application, e.g., which may be executable to performat least a portion of the methods described herein. Additionally, thememory medium may store a programming development environmentapplication (or developer's tools application) used to createapplications, e.g., for execution by the target device 150. The memorymedium may also store operating system software, as well as othersoftware for operation of the computer system. Various embodimentsfurther include receiving or storing instructions and/or dataimplemented in accordance with the foregoing description upon a carriermedium.

As also shown in FIG. 1A, the target device 150 may include a display,which may be operable to display graphics provided by an applicationexecuting on the target device 150. The application may be any ofvarious applications, such as, for example, games, internet browsingapplications, email applications, phone applications, productivityapplications, etc. The application may be stored in a memory medium ofthe target device 150. The target device 150 may include a centralprocessing unit (CPU) and a graphics processing unit (GPU) which maycollectively execute the application. For example, the CPU may generallyexecute the application as well as a graphics framework (e.g., OpenGL,DirectX, etc.) and graphics driver which may handle any graphics callsor commands that are provided by the application during execution. Thegraphics driver may in turn provide GPU commands to the GPU, which mayexecute these commands to provide display capabilities for theapplication. As used herein, a “graphics application” refers to anapplication which provides graphics commands for displaying graphics ofthe application on a display. In other words, the term “graphicsapplication” refers to a software application that, when executed,causes the display of various graphics on a display.

The memory medium of the target device 150 may also store one or moreprograms for implementing embodiments described herein. For example, thememory medium of the target device 150 may store a program for capturingand encoding graphics commands received from the application. The memorymedium of the target device 150 may also store a program for playingback a stream of graphics commands, e.g., which may be provided from thecomputer system 100. Further, the memory medium of the target device 150may store a program for performing measuring or monitoring (e.g., atdifferent levels of detail) of the application when it is executing onthe target device 150. In further embodiments, the programs may bestored on the computer system 100 and may be read onto the target device150 for execution.

FIG. 1B illustrates a system including the computer system 100 that iscoupled to the target device 150 over a network 125. The network 125 canbe any of various types, including a LAN (local area network), WAN (widearea network), the Internet, or an Intranet, among others. In general,the computer system 100 and the target device 150 may be coupled in anyof various manners, such as wired (e.g., over a serial bus, such as USB,Ethernet, Internet, etc.) or wireless (e.g., WLAN, Bluetooth, IR, etc.).

FIG. 1C illustrates a system where the host computer system 100 iscoupled to the target device 150 as well as another target device 175.As shown, the target device 175 may be a different type of target devicethan the target device 150. In one embodiment, the application may beexecuted on both of the target device 150 and 175. For example, graphicscommands captured on the target device 150 may be modified for testingand executed on one or both of the target devices 150 and 175.Accordingly, testing results and/or suggestions may be provided that aregeneric and/or specific to a particular target device. Thus, the resultsand/or testing may vary among different types of target devices. Furtherdetails regarding testing and capturing graphics commands are providedbelow.

FIGS. 1D and 1E illustrate systems where a computer system may be thetarget device. In FIG. 1D, the computer system 100 may be the targetdevice as well as the host device. In this embodiment, the computersystem 100 may execute both the target application and the controlprogram, thus effectively operating as both the host and target device.Alternatively, in FIG. 1E, a different computer system 190 may be thetarget device. The two computer systems 100 and 190 may be coupled overthe network 125 as shown, or may be coupled directly, as desired.

FIGS. 2A-2B—Exemplary Block Diagrams of Graphics System Hardware

FIGS. 2A and 2B are block diagrams of embodiments of target devicehardware implementing a graphics system. It is noted that FIGS. 2A and2B are simplified block diagrams, wherein various components that wouldnormally be present, but which are not necessary for an understanding ofthe invention, are omitted for simplicity and clarity.

More specifically, FIG. 2A illustrates one embodiment of a hardwarearchitecture of a target device computer system such as 150, 175, 100 or190. As shown, the CPU 202 and CPU memory 208 may be coupled together(e.g., over a system bus) and GPU 204 and GPU memory 210 may also becoupled together. The CPU 202 and GPU 204 (and their correspondingmemories) may be coupled via bus interface 206. For example, in oneembodiment, the GPU 204 and GPU memory 210 may be implemented as a videosystem having a different system interface than the CPU 202 and CPUmemory 208. For example, the GPU 204 and GPU memory 210 may beimplemented as a video card that is plugged in to a slot of the computersystem 100 or 190. The video card may be implemented as a PCI, PCIe,AGP, etc. card. Accordingly, bus interface 206 may interface with thesystem bus of the CPU 202 and the bus of the video card. The targetdevice, 150, 175, 100 or 190 would also include display logic (notshown) as well as various other logic.

FIG. 2B illustrates an alternative embodiment of a hardware architecturethat may be implemented by target device 150 or 175. In thisarchitecture, the CPU 202 and GPU 204 may be coupled over a system busand may share a common or unified memory 258 (although separate memoriesare envisioned). Additionally, a display block 260 may be coupled tomemory 258 and GPU 204 for displaying various images on the display ofthe target device 150 and 175. This implementation may apply to deviceswhose internal hardware are all or mostly provided within a singleintegrated chip, e.g., as a system on a chip (SOC).

It should be noted that the above hardware architectures of the graphicssystem are exemplary and are provided for illustration purposes only.Thus, various modifications (e.g., of blocks or connectivity) resultingin different hardware architectures are envisioned.

FIG. 3—Exemplary Block Diagram of Software Architecture

FIG. 3 is a block diagram of one embodiment of a software architecturethat may implement various embodiments described herein.

As shown in FIG. 3, the host 100 may execute a development environmentor control application 410. The development environment 410 may be usedto develop applications for execution on the target device 150. Thedevelopment environment 410 may also control execution of a developedapplication 450, a playback application 455, a measurement application460, etc. that may be executing on the target device 150.

As also shown in FIG. 3, the target device 150 may execute a variety ofprograms, including application 450, measurement application 460,playback application 455, graphics framework 470, and graphics driver480. While this diagram largely shows programs that are executed by theCPU of the target device 150, note that the GPU of the target device 150may also execute programs, e.g., shaders, that may be provided by theapplication 450.

In more detail, the application (or graphics application) 450 may be anapplication that is under development or testing, e.g., within thedevelopment environment 410. For example, a developer may be developingthe application on the host 100 for ultimate deployment and execution onthe target device, and may periodically need to test or debug theapplication while it is executing on the target device 150.Correspondingly, the development environment 410 may be used to deploythe application to the target device 150 for execution and testing.

The development environment 410 may also deploy other software to thetarget device 150 to assist in developing the application 450, e.g.,once the developer has designated that the target device 150 is used fordevelopment of the application 450. For example, the developmentenvironment 410 may deploy the measurement application 460 which maymeasure (or monitor) the execution of the application 450 on the targetdevice 150. In some embodiments, as described below, the measurementapplication 460 may be operable to measure (or monitor) at a first levelof detail (e.g., a first level of intrusion or cost) and at a secondhigher level of detail (e.g., at a second higher level of intrusion orcost). For example, measuring at the first level of cost may providefewer details than the second level of cost, but may not impactperformance of the execution of the application 450. On the other hand,measuring at the second level of cost may garner more detailedinformation regarding the execution of the application 450 (e.g., thegraphics performance of the application 450), but may impact theperformance of the execution of the application 450.

As described below, when the measurement application 460 operates in thesecond level of cost, it may intercept and record graphics commandsprovided by the application 450 to the graphics framework 470, as shown(referred to as “graphics framework commands”). However, in alternateembodiments, the measurement application 460 may be configured tointercept and record commands at other times, e.g., between the graphicsframework 470 and the graphics driver 480 (referred to as “graphicsdriver commands”) or even commands from the graphics driver 480 to theGPU 204 (referred to as “GPU commands”), as desired. Further, in oneembodiment, rather than being coupled to both origin and destinationprograms in the manner shown, the measurement application 460 may beinterposed between the origin and destination programs (or blocks).

These intercepted commands may be encoded as a command stream which maybe used for further testing and analysis, as described below. Forexample, the development environment 410 may deploy the playbackapplication 455 to the target device 150, which may be configured toplay back the intercepted (recorded) graphics commands, e.g., to thegraphics framework 470 as shown, or to other blocks, depending on wherethe commands were originally intercepted. As discussed further below,the playback application 455 may be configured to play back variousmodified versions of the recorded graphics commands to “try out” variouspossibilities for improving application execution performance. Note thatthe playback application 455 and the measurement application 460 may bethe same application.

The graphics framework 470 may be any of various types of graphicsframeworks, e.g., various versions of openGL (including openGL forembedded systems (ES)), DirectX, etc. The graphics framework 470 mayreceive API calls from the application 450 for performing graphicsframework functions. In turn, the graphics framework 470 may providecommands to the graphics driver 480, which may also be executing on thetarget device 150. Finally, the graphics driver 480 may provide GPUcommands to the GPU. The CPU executing the graphics framework 470 andthe graphics driver 480, along with the GPU may form a graphicspipeline, such as those embodiments described in FIGS. 4A and 4B below.

Note that the above software architecture is exemplary only and othervariations and modifications are envisioned. For example, in someembodiments, the graphics framework 470 may not be necessary and/or maybe implemented as part of the application 450 rather than being aseparate executable.

FIGS. 4A and 4B—Exemplary Graphics Pipeline

FIGS. 4A and 4B illustrate exemplary graphics pipelines. Moreparticularly, FIG. 4A illustrates an OpenGL embedded system (ES) 2.0pipeline and FIG. 4B illustrates an OpenGL embedded systems (ES) 1.1pipeline, e.g., which may be suitable for a target device 150, such as amobile device. However, the pipelines of FIGS. 4A and 4B may also beimplemented on a computer system such as computer system 100, e.g., withfurther modifications. For example, a typical OpenGL pipeline may beused for the computer system 100 (rather than an ES pipeline). Thepipelines of FIGS. 4A and 4B may be implemented using the graphicssystem of FIG. 2A or 2B and may also interact with the softwarearchitecture of FIG. 3. For example, the pipeline of FIGS. 4A and 4B maybe implemented as software executing on the CPU and/or GPU processes.Note that the GPU may execute various software on the GPU to performportions of the graphics pipeline and/or may include dedicated hardwarefor performing those portions, as desired.

In the graphics pipeline of FIG. 4A, the pipeline may begin with vertexdata in 402. The vertex data may specify the vertices of the graphicsdata to be rendered. In one embodiment, the vertex data may include dataabout polygons with vertices, edges and faces that constitute an entirescene.

In 404, the vertex data of 402 may be processed by a vertex shader. Moreparticularly, the vertex shader may be run for each vertex, e.g., by theGPU. This process may transform each vertex's 3D position in virtualspace to the 2D coordinate at which it will appear on the display. Thevertex shader may manipulate various properties, including position,color, texture coordinate, etc. As shown, the vertex shader 404 may beinformed by texture data 416 and/or shader uniform data 418.

In 406, primitives may be assembled from the vertices output from 404.For example, in this stage vertices may be collected and converted intogeometric shapes, e.g., triangles.

In 408, the primitives may be used in rasterization. More particularly,the primitives from 406 may be filled with pixels or fragments.

In 410, the fragment shader (e.g., executed by the GPU) may add texturesand final colors to the fragments. Fragment shaders may typically takeinto account scene lighting and related effects, such as bump mappingand color toning. As shown, the fragment shader may be informed bytexture data 416 and shader uniform data 418.

In 412, various per-fragment operations may be performed. For example,the operations may combine the final fragment color, its coverage,and/or degree of transparency with the existing data stored at theassociated 2D location in the frame buffer to produce the final colorfor the pixel to be stored at that location.

In 414, the data may be stored in physical memory which holds the actualpixel values displayed on the screen. The frame buffer memory may alsostore graphics commands, textures, and/or other attributes associatedwith each pixel. This data may be used to output the final image to thedisplay.

FIG. 4B illustrates an abbreviated pipeline that may be more appropriatefor embedded systems. As shown, is the pipeline includes vertex data402, per-vertex operations 454 (similar to vertex shader 404), primitiveassembly 406, rasterization 408, per-fragment operation 412, and framebuffer 414. This pipeline does not utilize the fragment shader 410 orthe shader uniform data 418. Additionally, the texture data is onlyutilized by rasterization 408 rather than by shaders, as in FIG. 4A.

Thus, FIGS. 4A and 4B illustrate exemplary graphics pipelines that maybe utilized in embodiments described herein. However, other, differentgraphics pipelines are envisioned.

FIG. 5—Analyzing Execution of an Application on a Target Device

FIG. 5 illustrates a method for analyzing an application executing on atarget device. The method shown in FIG. 5 may be used in conjunctionwith any of the computer systems or devices shown in the above Figures,among other devices. In various embodiments, some of the method elementsshown may be performed concurrently, in a different order than shown, ormay be omitted. Additional method elements may also be performed asdesired. As shown, this method may operate as follows.

In 502, a target device may be coupled (or paired) to a host device. Asindicated above, the target device and host device may be coupledtogether in a variety of ways, e.g., directly via a wired or wirelessconnection, or indirectly, over a network (e.g., an Intranet or theInternet).

In 504, an application may be executed on the target device. Forexample, the application may be developed using a developmentenvironment program executed by the host device. The user (e.g., thedeveloper) may compile the application for execution and/or deploy theapplication to the target device. As indicated above, the developmentenvironment program may also deploy other programs to the target device,e.g., measurement programs, playback programs, etc.

Once deployed on the target device, the development environment programmay initiate execution of the application (and/or any other programs,such as those described above) on the target device, e.g., by sending anexecution command to the target device. Thus, in 504, the applicationmay be executed by the target device.

In 506, during or after the execution of the application, the targetdevice may provide application execution information to the host device.The application execution information may be any of a variety ofinformation. For example, the application execution information mayinclude information gathered by a measurement application executing onthe target device while the application is executing. For example, themeasurement application may gather CPU load information, GPU loadinformation, and/or other information. In further embodiments, themeasurement application may intercept and record graphics commandsprovided by the application. The application execution information mayinclude those graphics commands (e.g., encoded in a bitstream). Asdescribed in FIGS. 6 and 7 below, the application execution informationmay be gathered in response to input from the host device, e.g., inresponse to explicit user input, or may be gathered automatically basedon various detected conditions or “triggers”.

In 508, the execution information may be analyzed. For example, theexecution information may be analyzed to determine any execution issuesor performance bottlenecks of the application. In some embodiments,these issues may be particularly identified in relation to graphicsperformance. For example, the execution information may be analyzed todetermine which portion of the graphics pipeline is causing aperformance bottleneck during execution of the application. Theexecution information may also be analyzed to determine if the CPU orGPU is limiting the graphics performance of the application. Moredetails on this particular CPU/GPU analysis is provided in FIG. 9 below.

In some embodiments, further analysis (e.g., similar to 506 and 508) maybe performed repeatedly (as described in FIG. 8 below) in order toidentify or determine various performance issues (e.g., graphicsbottlenecks) of the application.

Finally, in 510, based on the analysis, suggestions may be provided toincrease performance (or remove bottlenecks) of the application. Forexample, where a particular texture is determined to cause a performancebottleneck in the graphics pipeline, the method may provide a suggestionto use a more compact or compressed version of the texture or even adifferent texture, e.g., a less complex (e.g., lower resolution)texture. Thus, the suggestions may help a developer create a morestreamlined application, particularly with respect to graphics of theapplication. Note that the suggestions may be specific to increasingperformance of the application itself, or may be specific to increasingperformance of the application when specifically executing on aparticular target device. For example, improvements may not necessarilybe required on a high performance system, but may be required when theapplication is executing on a lower performance system. Additionally, ifa particular target device has a lower texture buffer, the suggestionsmay be specific to improving performance because of that particulartexture buffer on the particular target device. Thus, the suggestionsmay be generic with respect to target devices (but still specific to theapplication) or target device specific, as desired.

The analysis and provision of suggestions may be performed by thedevelopment environment program executing on the host device, or byanother application, as desired. More specific methods and examples areprovided below.

FIG. 6—Triggering High Cost Measurement of an Application Executing on aDevice

FIG. 6 illustrates a method for triggering high cost measurement of anapplication executing on a target device. The method shown in FIG. 6 maybe used in conjunction with any of the computer systems or devices shownin the above Figures, among other devices. In various embodiments, someof the method elements shown may be performed concurrently, in adifferent order than shown, or may be omitted. Additional methodelements may also be performed as desired. As shown, this method mayoperate as follows.

In 602, execution of the application by the target device may begin,similar to 504 above.

In 604, low cost (e.g., non-intrusive or less intrusive) measurement ofexecution of the application may be performed. More particularly, thelow cost measurement may measure graphics performance of theapplication. As indicated above, the low cost measurement may beperformed by a measurement application that is deployed (e.g., stored inmemory) on the target device. Thus, the measurement program may executealong with the application on the target device and may perform low costmeasurement of the application executing on the target device. The lowcost measurement may be performed at a level that does not substantiallyinterfere with the execution of the application. For example, in someembodiments, the low cost measurement may not cause a perceptible (by auser) change in graphics performance of the application. In oneembodiment, during low cost measurement, the application may executewith at least 90%, 95%, 98%, 99%, etc. of the resources or processingtime than it would have had if the low cost measurement were not beingperformed. In other embodiments, the low cost measurement may cause aperceptible change in graphics performance of the application, but thisamount of perceptible change is quite a bit less than the amount ofperceptible change caused by the high cost measurement.

In one embodiment, low cost measurement may measure the followingparameters: frame rate of the graphics application executing on thetarget device, percentage CPU utilization (or CPU load), percent of CPUtime in graphics framework versus not, percent of CPU time spent waitingfor the GPU, percentage of GPU utilization (e.g., average tilerutilization percent and average render utilization percent), GPU powerconsumption, etc. The low cost measurement may also measure dataspecifically from the driver. For example, the driver may be configuredto record how long it spends in specific sections of the graphicspipeline, e.g., including frame presentation wait time (time spentwaiting for the system to present the frame buffer and provide anotherrender buffer), texture upload time (time spent uploading texture datato the GPU), state validation time (time spent validating that thegraphics framework state is valid), vertex copy time (time spentconverting and copying vertex data to the GPU), etc. These times may berecorded using counters and may be used to determine relativepercentages of time spent on these sections of the driver.

In some embodiments, during the low cost measurement, low costmeasurement data may be provided from the target device to the hostdevice. The provided low cost measurement data may include all or aportion of the parameters being measured during low cost measurement.Additionally, the low cost measurement data may be provided for displayby the host device, e.g., for presentation to a user or developer usingthe host device.

In 606, a trigger may be received to perform high cost measurement ofthe application. The trigger may be provided via a variety ofmechanisms. For example, in one embodiment, the target device may becoupled to a host device during execution. During execution, the hostdevice may receive input from a user (e.g., the developer) to begin highcost measurement. For example, the user may be monitoring graphicsperformance of the application on the target device or monitoringprovided low cost measurement data. The user may then decide to manuallyinvoke the high cost measurement of the application, e.g., afterperceiving a performance issue in graphics performance of theapplication. Accordingly, the host device may provide a signal to thetarget device to begin the high cost measurement of the application,thereby triggering the high cost measurement of the execution of theapplication. In alternate embodiments, the user input may be provideddirectly to the target device instead of via the host device.

Alternatively, or additionally, the trigger may be detected/provided inan automatic fashion. For example, the device and/or the host may havecertain conditions that, when detected during application execution,will initiate the high cost measurement of the application. Theseconditions may be related to detected performance issues. For example,the conditions may correspond to a graphics performance issue duringexecution of the application. In one embodiment, the conditions may bebased on the parameters being measured by the low cost measurement. Forexample, the trigger may be a threshold frame rate or a threshold changein frame rate of the graphics of the application on the target device.The trigger may also be based on GPU load, CPU load, CPU graphics load(e.g., CPU load devoted to the graphics driver, framework, or pipelinein general), etc. For example, the trigger may be based on a thresholdchange (e.g., increase) in one or more of GPU utilization, GPU blockutilization (e.g., tiler, rasterizer, etc.), graphics framework CPUutilization (e.g., in general or specific to a particular task, such asvertex copy, state validation, texture upload, etc.), GPU power usage,etc.

The condition may be automatically determined by the host and/or thetarget device. When determined by the target device, the target devicemay simply utilize a condition detected by the low cost measurementinformation to automatically trigger the high cost measurement. Whendetermined by the host device, the target device may continually orperiodically provide the low cost measurement information to the hostdevice, which may then analyze that information (e.g., compared toprevious measurement information) to detect the condition. Accordingly,upon detection of the triggering condition, the host device may providean interrupt or other signal to the target device to initiate andperform the high cost measurement of the application.

In one embodiment, rather than capturing all of the low cost measurementdata described above throughout execution of the application, only theconditions for triggering may be monitored. Upon reaching a triggeringcondition (the trigger), a snapshot of the performance of theapplication may be gathered, including all or a portion of theparameters described above in the low cost measurement portion (e.g.,CPU load, GPU load, GPU power, etc.). Once this snapshot has beenrecorded, the high cost measurement may be triggered as described. Bycapturing this snapshot, this information may provide a baselineperformance of the graphics application for the period of time when thehigh cost measurement is being performed, since the high costmeasurement impacts performance of the execution of the application. Inother words, the snapshot may provide the most accurate executionconditions for the period of time that high cost measurement isperformed since it is directly preceding that time and the low costmeasurement does not significantly impede application execution. Notethat this snapshot may also be recorded after completion of the highcost measurement.

In 608, in response to the trigger to begin high cost measurement, highcost measurement (e.g., intrusive measurement) of the application may beperformed. Unlike some embodiments of low cost measurement, as indicatedabove, high cost measurement may typically impact performance of theexecution of the application. For example, the high cost measurement maycause a noticeable (by a user) impact to performance of the application.The high cost measurement may be used to allow for more extensivetesting of the application. For example, the high cost measurement maycontinually measure more detailed information of execution of theapplication (e.g., in a full profile mode). The high cost measurementmay also record all graphics commands (e.g., and associated graphicsdata) for later analysis and playback (e.g., in a record mode). FIG. 7describes one embodiment of a method for performing high costmeasurement of an application executing on a target device. The low costmeasurement and high cost measurement may be performed by the samemeasurement program or different measurement programs, as desired.

In 610, before, during, and/or upon completion of the high costmeasurement, measurement information of execution of the application maybe provided to the host device, e.g., for further analysis, as in 508above. The measurement information may include the high cost measurementinformation and/or the low cost measurement information, as desired. Forexample, the low cost measurement information may be providedperiodically during low cost measurement and the high cost measurementinformation may be provided periodically during the high costmeasurement. Alternatively, the measurement information may be providedafter completion of each stage, or after completion of all measurement.

By generally performing low cost measurement, and only performing highcost measurement when certain undesirable conditions are detected, amore accurate performance of the application executing on the targetdevice is measured. Were an intrusive measurement always performed,various problems might arise that are avoided using the method describedabove. For example, intrusive measurement may not perturb theapplication evenly across different units or stages of the graphicspipeline, which can disguise a performance bottleneck in one area, e.g.,by creating another in a different area. Additionally, since thegraphics of an application can achieve a maximum frame rate (wheregraphics improvements become moot), the graphics rate can be quantizedby the refresh rate interval. Accordingly, when such perturbationoccurs, it may be magnified due to the quantization caused by therefresh rate interval. Thus, by performing low cost measurement totrigger high cost measurement, these potential problems may be avoided.

Note that the method of FIG. 6 may be performed repeatedly throughoutexecution of the application. For example, low cost monitoring may bereinitiated after the high cost monitoring is terminated, and the highcost monitoring may be retriggered at a later point in execution of theapplication. This process may continue until the application isterminated. Alternatively, upon completion of the high cost monitoring,the application may be automatically terminated.

FIG. 7—Performing High Cost Measurement of an Application Executing on aDevice

FIG. 7 illustrates one embodiment of a method for performing high costmeasurement of an application executing on a target device. The methodshown in FIG. 7 may be used in conjunction with any of the computersystems or devices shown in the above Figures, among other devices. Invarious embodiments, some of the method elements shown may be performedconcurrently, in a different order than shown, or may be omitted.Additional method elements may also be performed as desired. As shown,this method may operate as follows.

In 702, an indication to perform high cost measurement of an applicationexecuting on a target device may be received. For example, theindication may be received as a trigger as described in FIG. 6 above.

In 704, a graphics command may be intercepted. For example, the graphicscommand may be intercepted by a measurement application (or graphicscommand capture application) executing on the target device. In oneembodiment, the graphics command may be provided from the applicationand may be intended for reception by a graphics framework. In thisembodiment, the graphics command may be an API call from the applicationto the graphics framework. Accordingly, the measurement application mayintercept the command before it is provided to the graphics framework.In one embodiment, the graphics command may be particularly interceptedby an interpose library (e.g., the measurement application or a portionof the measurement application) that may be injected into theapplication executing on the target device. Accordingly, this librarymay intercept all graphics framework functions and other platform APIsand gather execution state information of the application and/or targetdevice when graphics commands are issued.

Alternatively, the graphics command may be intercepted between thegraphics framework and a graphics driver executing on the target device.In further embodiments, the graphics command may be intercepted betweenthe graphics driver and the GPU, although those commands may begenerally referred to as GPU commands rather than graphics commands.

In 706, the graphics command may be stored in a first buffer. Thegraphics command may be stored in the buffer as originally received, ormay be modified. For example, where the graphics command referencesadditional graphics data (e.g., texture data), which is usually done viaa pointer to a memory address, the pointer may be modified to referencea file name or different storage path (e.g., via a URI or URL). Furtherdetails regarding storage of the graphics data is provided in 708 below.Additionally, the graphics command may be encoded in a different format(e.g., a bitstream) than the original format.

Further, additional data (or additional information) regarding thegraphics command (or associated with it) may also be determined andstored in the first buffer. For example, the additional data may bederived from the graphics command and/or from the graphics system (e.g.,the state of the graphics system, the GPU load, the CPU load, or anyother information). Thus, the additional data may be determined afterthe graphics command is intercepted and includes new data that was notincluded in the graphics command originally.

The additional data may be any of a variety of flags or commands. Forexample, the additional data may include a “no-execute” flag, which mayindicate that the graphics command should not be executed (or should beignored) by any future playback applications, but possibly not foranalysis. The additional data may include an “inserted” command andassociated flag, which may indicate that the inserted command wasgenerated by the measurement application and should be executed byplayback applications, but possibly not processed by other systems(e.g., during analysis). For example, the inserted command may addfunctionality, such as creating a new surface or other graphicsfunction.

The additional data may include a “no-trace” flag, which may indicatethat the command should not be traced, printed, or displayed to the userby any system, but should otherwise be considered for analysis. Theadditional data may include a “backtrace”, e.g., a set of bytesrepresenting the execution state of the application (e.g., of auser-space thread of the target device) at the time the command wasbeing executed. The backtrace may be encoded in a platform independentmanner. The additional data may include “trace events”, e.g., a set ofbytes including a platform independent header followed by a platformdependent structure (e.g., bitstream) encoding various events, state,performance counters or data current at the time the command was beingexecuted. In one embodiment, this could be a bitfield indicating thatcertain events have occurred in the graphics framework or driver betweenthe time of the previously intercepted command and the current command.Further, the additional data may specify performance data (e.g.,graphics performance data) of the application (e.g., framerate, CPUload, GPU load, etc.).

The additional data may include metadata for the command, e.g., a set ofbits indicating the nature of the command. In one embodiment, this couldbe a set of bits indicating if the command is a C function or anObjective-C method. This metadata may also specify an indicator forwhether the format is big endian or little endian, bit width or otherbit formats, etc. This additional data may be useful in ensuring thatthe graphics commands are later executable by target devices other thanthe one from which the commands are being captured (e.g., which may havedifferent bit formats or lengths).

The additional data may also indicate type information, including, forexample, core type (e.g., storage characteristics of the data, char,float, double, etc.) and semantic type (e.g., number, constant, symbols,etc.). By storing this typing information, rich tracing may be moreeasily performed. For example, a value can be converted to a string(e.g., where the value is a constant), which may allow the value to belater printed.

The additional data may also indicate that a particular value (e.g.,associated with the graphics command) is a variable. In this embodiment,rather than storing the value, a new variable may be created.Accordingly, when the graphics command is re-executed (e.g., in playbackduring analysis), the returned value may be different, but may berecognized as a variable, thereby ensuring that the value of thatvariable is repeatable in later uses (e.g., by later graphics commands)during playback.

The additional data may include a thread ID, e.g., a numericalidentifier for the system thread in which the command was executing. Theadditional data may include a timestamp, e.g., a timestamp relative to acapture clock or timer (e.g., of the measuring application) indicatingthe time at which the command was executed or captured (e.g., encoded).The additional data may include the amount of time (e.g., innanoseconds) that the command took to execute. The additional data mayinclude a graphics framework error (e.g., an OpenGL error), which mayindicate the current frameworks error (e.g., part of the graphicsframework state machine) at the time the command was being executed.

In 708, if the graphics command referenced any graphics data, thatgraphics data may be stored in a second buffer (although in otherembodiments, the data may be stored in the first buffer with thegraphics command and additional data). The graphics data may includetextures referenced by the graphics command. For example, the graphicscommand may reference a texture stored in memory, e.g., via pointerspointing to specific memory address(es). Accordingly, the graphics datamay be retrieved from the memory and stored in the second buffer. Insome embodiments, any graphics data may be stored as individual files inthe second buffer with corresponding names. For example, a referencedtexture may be stored as a single file per texture. Alternatively, allof the graphics data may be stored in a graphics data file, as desired.As indicated above, the graphics command may be modified when stored inthe first buffer to reference the new location or file name of thegraphics data, rather than the original pointer to the memoryaddress(es).

In some embodiments, if the graphics data has been previously copied tothe second buffer (or previously copied in general), a reference to thegraphics data may be stored rather than re-storing the graphics dataitself, thereby saving memory space and transfer time. Thus, redundantgraphics data may not be duplicated in the second buffer (or for thegraphics commands in general).

In 710, the original graphics command may be provided to itsdestination. In some embodiments, the graphics command may be providedfrom the measuring application to the destination (e.g., the graphicsframework), e.g., in embodiments where the measuring applicationcaptured the graphics command and prevented it from being provided tothe destination. Where the graphics command was captured between otherentities (e.g., between the framework and the driver, it may be providedto the driver. Alternatively, the graphics command may have beenoriginally delivered to the destination, but the measuring applicationmay have copied the command prior to delivery. In this embodiment, 706and 708 may operate on the copy of the graphics command.

704-710 may be performed one or more times until the first and/or secondbuffers are full or until the high cost monitoring is completed.

In 712, the multiple, stored graphics commands and associated additionaldata from the first buffer and the accumulated graphics data from thesecond buffer may be provided to the host device, e.g., once the buffersare full. In some embodiments, the graphics commands and additional datamay be encoded, e.g., as a bit stream. The graphics commands may havebeen encoded in 706 or may be encoded at the time of transfer to thehost device.

The process of 704-712 may be performed multiple times within the highcost measurement. The commands, additional data, and graphics data maybe aggregated as a file or directory, which may be used for latertesting and analysis, as described in FIG. 8 below. The commands andadditional data (and possibly the graphics data) may be aggregated as acommand stream. In embodiments where the command stream is encoded, theencoded data may be a bit stream that includes a function identifier anddescriptions of the arguments and values. For example, for values thatare sensitive to the platform (e.g., pointer size, endian character,etc.), the entire command stream may be marked, such that it can beconverted to the appropriate platform for later execution, depending onthe target device. Thus, the command stream may be executable forplayback on any of various different platforms regardless of theoriginal platform on which it was recorded.

Finally, in 714, an indication to terminate the high cost measurementmay be received, e.g., from the host device.

As indicated in the descriptions of FIG. 6, the method of FIG. 7 may beperformed a once or multiple times during the execution of theapplication, as desired.

FIG. 8—Determining Performance Issues of an Application Executing on aDevice

FIG. 8 illustrates a method for determining performance issues of anapplication executing on a target device. The method shown in FIG. 8 maybe used in conjunction with any of the computer systems or devices shownin the above Figures, among other devices. In various embodiments, someof the method elements shown may be performed concurrently, in adifferent order than shown, or may be omitted. Additional methodelements may also be performed as desired. As shown, this method mayoperate as follows.

In 802, a command stream and performance data may be received based onan application executing on a recording target device, e.g., asdescribed in FIGS. 6 and 7 above.

In 804, the command stream may be provided to a target device forreplay. The target device may be the same target device that the commandstream was originally recorded by, or another target device, as desired.In some embodiments, where the target device is the same as the devicewhich recorded the commands of the command stream, the target device mayalready have the command stream and may not need to be transferred tothe target device for replay.

In one embodiment, rather than using the performance data received fromthe recording target device executing the application, the commandstream may be replayed on the target device to determine a baselineperformance of the command stream. Since the command stream may notconstitute the whole of the application executing on the target device(because other processes are executed during the application'sexecution), this baseline performance may be more useful for comparisonwith executions of modifications to the command stream. Thus, in oneembodiment, the command stream may be first executed on the targetdevice and performance may be gathered on that execution of the commandstream for comparison to later performance data.

The command stream may be executed in a variety of manners. For example,the command stream may include initial state data that may be used torecreate the initial graphics state of the application when the commandstream was initially recorded. Accordingly, the target device mayinitiate the graphics framework, graphics driver, graphics pipeline,memory mediums, etc. to replicate the initial state when the commandstream was recorded. The commands of the command stream may then beexecuted to generate the performance data. In some embodiments, thecommands may simply be executed a single time to generate theperformance data.

However, it may be desirable to execute the command stream multipletimes to reach a steady state in execution, and then generateperformance data of execution of the command stream once that steadystate has been reached. In order to execute the command stream multipletimes, the command stream may also store final state data that indicatesthe final graphics state of the application when the command stream wasfinished recording. This final state information may be used to generatea set of commands to revert the final state to the initial state.Accordingly, the graphics system may be initiated at the initial state,the command stream may be executed, the set of commands to revert thefinal state to the initial state may be executed, and the process may berepeated until a steady state is reached. As indicated above, once thissteady state is reached, the performance data of the command stream maybe recorded. This performance data may be based on a single execution ofthe command stream or may be based on multiple executions of the commandstream (e.g., an average or median execution of the command stream). Bygathering performance data over multiple executions of the commandstream, more accurate performance data may be gathered.

In 806, a modification to the command stream may be generated andprovided to the target device. The modification to the command streammay be used to test a particular operation or portion of the graphicspipeline to determine if that operation is causing a performance issueor bottleneck for the application. Stated another way, modifications tothe command stream may be made to “try out” different executionscenarios (or experiments) in order to attempt to isolate various“causes” of performance issues. More detailed descriptions regarding themodifications to the command stream to perform testing are providedimmediately after descriptions of FIG. 8.

In further embodiments, in order to test a portion of the graphicspipeline, various switches may be modified in the graphics driver todisable parts of the graphics pipeline. For example, color writes may bedisabled for the entirety of the command stream or within a portion ofthe command stream (e.g., within a frame) using graphics driverswitches. These switches may be modified in addition to, or potentiallyinstead of, the modification to the command stream. For example, thegraphics driver switches may be used to disable one or more portions ofthe graphics pipeline via initial state data (thereby disabling theportion of the graphics driver for at least the beginning, andpotentially throughout the entirety, of the execution of the commandstream) or may be set during playback of the modified command stream. Insome embodiments, the switches may be set as playback commands withinthe command stream. Alternatively, or additionally, the switches may beset externally, e.g., as specified within a current test or experimentdefinition. When set externally, the switches may be enabled or disabled“on the fly” during execution of the modified command stream in 810. Insome embodiments, the host may provide instructions to the target devicefor setting these switches during execution of the modified commandstream, e.g., as initial state data, as part of an experimentdefinition, and/or as part of the modification to the command stream,among other possibilities.

In 808, the target device may modify the command stream (e.g., using thecommand stream player application) according to the modification to thecommand stream, to generate a modified command stream.

In some embodiments, rather than providing the command stream and thenproviding a modification to the command stream, a modified commandstream may be generated and provided to the target device. However,modifications to the command stream are generally much smaller than theentirety of the command stream itself, so, when there are multiple testsor modifications to the command stream to perform, it may generally bemore efficient to provide the original command stream and subsequentmodifications to the command stream. As used herein, provision of one ormore modified versions of the command stream may refer to provision ofthe modified command stream or simply provision of modifications to thecommand stream that may be used to generate modified command streams.

In 810, the target device may execute the modified command stream andmeasure the performance of execution of the modified command stream,e.g., using the low cost measurement of the measurement application. Theexecution of the modified command stream may be performed in a varietyof ways, as described above regarding the execution of the originalcommand stream in 804.

In further embodiments, rather than modifying the command stream andseparately executing the modified command stream (as in 808 and 810above), the original command stream may be modified “dynamically” or “onthe fly” using the modification to the command stream. However, even inthis embodiment, the target device is still executing the modifiedcommand stream, but simply has not created an entire new modifiedcommand stream from the original command stream. Thus, rather thancreating a modified command stream, the original command stream may bedynamically modified during execution using the modification received in806.

In 812, the performance data of the modified command stream may beprovided from the target device.

806-812 may be performed multiple times to perform different tests todetermine suggestions for overcoming performance issues of theapplication on the target device (e.g., to improve graphicsperformance).

In 814, the performance data may be analyzed. More particularly, theperformance data between the different modifications may be compared todetermine where there are performance issues or bottlenecks. Forexample, in one embodiment, the frame rates of each modified commandstream may be compared to determine relative increases or decreasesbetween each different modification. From this information relativeinformation can be identified among the different modifications todetermine where the performance issues are located (e.g., in thegraphics pipeline, or at lower levels, as desired). In one embodiment,the analysis may yield a cost per command or statement in the commandstream or even within a shader, which may be utilized to determine thebottleneck(s) of the application executing on the target application viathe modifications to the command stream.

In some embodiments, the results of the analysis may be provided fordisplay to the user, e.g., the developer, on the host device. Forexample, where a particular stage of the graphics pipeline has beenidentified as causing a performance bottleneck for the application, thisinformation may be provided to the user. In one embodiment, a graphicspipeline image may be displayed which highlights the portions of thepipeline that are causing a performance issue (e.g., as a heat map).Where particular operations or shaders are causing performance issues,these may be identified to the user as well.

In 816, based on the analysis, one or more suggestions may be provided.For example, the suggestions may be provided to the user on the hostdevice, e.g., in the development application executing on the hostdevice. The suggestions may be provided to the user in graphical ortextual manners, as desired. Exemplary suggestions and methods forproviding that information are provided in the section below as well asin the descriptions of FIGS. 10A-10E.

Exemplary Command Stream Modifications and Suggestions

The modifications to the command stream may be performed to create testsused to isolate any performance issues or bottlenecks in theapplication. In general, the tests may be initially used to isolate theproblem at higher levels all the way down to per operation, per command,or per texture issues. Correspondingly, the initial tests may identify aparticular stage in the graphics pipeline that is causing theperformance issue. From there, that stage may be further analyzed (e.g.,through analysis of more particular modifications to the command stream)to determine which specific operation, command, or texture within thatstage is causing the performance issue. Once the issue(s) areparticularly identified, one or more suggestions may be provided to theuser in order to overcome the identified issue(s).

In one embodiment, these modifications may be performed on a peroperation basis and may disable various fragment processing, disablecomplex texture filtering (e.g., by using a less costly filtering modewhen sampling from a texture, such as blinear filtering instead ofanisotropic filtering), disable texture compression, use a smallermipmap, reduce vertex information complexity, etc. In general, theexperiments may modify one or more of the following to determine agraphics bottleneck (e.g., after a stage has been identified as beingproblematic): a shader, a resource (e.g., textures, buffers, vertexdata, index data, etc.), or a graphics (e.g., framework) state (e.g.,texture filtering, alpha test state, depth test state). Thus, a shader(e.g., a vertex shader or fragment shader) may be replaced with asimpler shader (or nonexistent shader (identity version) to remove theshader's effect), a resource may be replaced with a simpler resource orremoved, and the state may be changed to a less intensive state, as someexamples. Additionally, the modifications may add or remove differentcommands from the command stream, as desired.

Based on these experiments, specific and tangible optimization (orimprovement) advice may be provided. For example, if the bottleneck isdetermined to be bound by the GPU performing texture lookup, thesuggestions may include:

Use compressed texture formats such as PVRTC2 and PVRTC4 wheneverpossible.

If can't use compressed textures, minimize texel size.

Use mipmapping whenever possible.

If the analysis determines that particular textures are responsible forthis then the suggestions may specifically advise these optimizationsfor the identified textures.

The following provides an overview of different modifications orexperiments that may be performed:

The present render buffer may be disabled, which may be performed on aper frame basis. Rendering in general may be disabled, which may beperformed on a per frame or per draw basis. Color writes may be disabledon a per frame or per draw basis. Depth writes may be disabled on a perframe or per draw basis. Stencil writes may be disabled on a per frameor per draw basis. Color masks may be overridden on a per frame or perdraw basis. Rasterization may be disabled on a per frame or per drawbasis. A minimal viewpoint may be used on a per frame or per draw basis.Cheap (or cheapest) texture filtering may be used on a per frame, perdraw, or per texture basis. Small textures may be used on a per frame,per draw, or per texture basis. Compressed textures may be used on a perframe, per draw, or per texture basis. Texture uploads may be disabledon a per frame, per draw, or per texture basis. Minimal vertex shaders(e.g., for OpenGL ES 2.0) may be used on a per frame, per draw, or pershader basis. Minimal fragment shaders (e.g., for OpenGL ES 2.0) may beused on a per frame, per draw, or per shader basis. Alpha testings(e.g., for OpenGL ES 1.1) may be disabled on a per frame or per drawbasis. Fragment discards (e.g., for OpenGL ES 2.0) may be disabled on aper frame, per draw, or per shader basis. Lighting (e.g., for OpenGL ES1.1) may be disabled on a per frame or per draw basis. Matrix palette(e.g., for OpenGL ES 1.1) may be disabled on a per frame or per drawbasis. Vertex fetching may be disabled on a per frame or per draw basis.Tiling may be disabled on a per frame or per draw basis. Interleavedvertex data may be forced on a per frame, per draw, or per vertex bufferbasis. Optimal vertex format may be forced on a per frame per draw, orper vertex buffer basis. Indexed drawing may be forced on a per frame,per draw, or per vertex buffer basis. Vertex buffer usage may be forcedon a per frame or per draw basis. Index buffer usage may be forced on aper frame or per draw basis. Redundant calls may be removed on a perframe basis.

The following provides one embodiment a more detailed account ofdifferent experiments that may be performed to isolate performanceissues of an application executing on a target device.

The following section describes exemplary overrides. Note that “->”means “replaced with”.

OverrideDisablePresent: presentRenderBuffer->glFlush (Except: ->glFinishon last frame of repeat set)

OverrideDisableColorWrites: Driver pipeline control:StageSkipColorWriteOut

OverrideTextureFiltering:

if minFilter=NEAREST or LINEAR

-   -   minFilter=NEAREST

else

-   -   minFilter=NEAREST_MIPMAP_NEAREST

magFilter=NEAREST

OverrideUseSmallTextures(all): For all textures:

Override glTexImage2D/glCompressedTexImage2D to load two levels of RGBAbyte data (level 0: 2×2, level 1: 1×1)Ignore calls to glTexSubImage2D and glCompressedTexSubImage2D

OverrideColorMask: ColorMaskRed/Green/Blue/Alpha=False/False/False/False

OverrideDisableRendering (OpenGL ES2.0 Only):

Driver pipeline control: StageSkipRender

OverrideViewport: viewport=(−40, −40, 0, 0)

OverrideShaderSource:

glShaderSourceARB for vertex shader: source->“{gl_Position=vec4(0.0,0.0, 0.0, 1.0);}”

glShaderSourceARB for frag shader: source->“{gl_FragColor=vec4(1.0, 1.0,1.0, 1.0);}”

OverrideDisableAll (OpenGL ES 2.0 Only):

Driver pipeline control: SkipAll

OverrideDisableES1VertexShading:

Disable NORMALIZE, RESCALE_NORMAL

Disable LIGHTING

Force TEXTURE_MATRIX to Identity

Disable MATRIX_INDEX_ARRAY_OES, WEIGHT_ARRAY_OES

The following describes exemplary experiments that may be performedusing the above overrides:

ExperimentDontFetchShadeVertex (OpenGL ES 2.0 Only):

Active overrides:

-   -   OverrideDisableAll    -   OverrideDisablePresent

ExperimentDontShadeVertex (OpenGL ES 1.1 Only):

Active overrides:

-   -   OverrideDisableES1VertexShading    -   OverrideViewport    -   OverrideDisablePresent

ExperimentDontTile:

Active overrides:

-   -   OverrideViewport    -   OverrideDisablePresent

ExperimentDontRender (OpenGL ES 2.0 Only):

Active overrides:

-   -   OverrideDisableRendering    -   OverrideDisablePresent

ExperimentDontShadeFragment (OpenGL ES 2.0 Only):

Active overrides:

-   -   OverrideColorMask    -   OverrideDisableColorWrites    -   OverrideDisablePresent

ExperimentUseSmallTextures(all) (Note: param=nil means all):

Active overrides:

-   -   OverrideUseSmallTextures(all)    -   OverrideTextureFiltering    -   OverrideDisableColorWrites    -   OverrideDisablePresent

ExperimentAllSimpleTextureFilter

Active overrides:

-   -   OverrideTextureFiltering    -   OverrideDisableColorWrites    -   OverrideDisablePresent

ExperimentDontWriteColor

Active overrides:

-   -   OverrideDisableColorWrites    -   OverrideDisablePresent

ExperimentDontPresent

Active overrides:

-   -   OverrideDisablePresent

The following provides exemplary experiments for the pipeline of GPUswith fragment shader capabilities that may apply to both OpenGL ES1 andOpenGL ES2 applications:

ExperimentDontFetchShadeVertex

ExperimentDontTile

ExperimentDontRender

ExperimentUseSmallTextures(all)

ExperimentAllSimpleTextureFilter

ExperimentDontShadeFragment

ExperimentDontWriteColor

ExperimentDontPresent

ExperimentDontFetchShadeVertex

ExperimentDontTile

ExperimentDontRender

ExperimentUseSmallTextures(all)

ExperimentAllSimpleTextureFilter

ExperimentDontWriteColor

ExperimentDontPresent

The following provides exemplary experiments for the pipeline of GPUswithout fragment shader capabilities:

ExperimentDontFetchVertex

ExperimentDontShadeVertex

ExperimentDontTile

ExperimentUseSmallTextures(all)

ExperimentSimpleTextureFilter

ExperimentAllSimpleTextureFilter:

ExperimentDontWriteColor:

ExperimentDontPresent

By performing the various experiments above, more detailed informationregarding the specific resources, shaders, or states that are causingthe performance issues may be gathered. Additionally, specificsuggestions for overcoming these issues may be presented to thedeveloper. This provides a much more efficient and helpful system inassisting a developer create an application that makes efficient use ofthe graphics system, and goes into a level of detail much greater than,for example, simply identifying that the GPU is limiting graphicsperformance. Thus, the experiments described above may allow a developerto more easily to determine per operation bottlenecks and solutions.

FIG. 9—Determining Whether the CPU is Limiting Graphics Performance

FIG. 9 illustrates a method for determining whether the CPU is limitingperformance (e.g., graphics performance) of an application executing ona target device. The method shown in FIG. 9 may be used in conjunctionwith any of the computer systems or devices shown in the above Figures,among other devices. In various embodiments, some of the method elementsshown may be performed concurrently, in a different order than shown, ormay be omitted. Additional method elements may also be performed asdesired. As shown, this method may operate as follows.

In 902, a graphics application may be executed on a target device,similar to 504 above.

In 904, during execution of the graphics application, performanceinformation of the CPU and GPU may be measured, e.g., by a measurementapplication executing on the target device. The measurement applicationmay be performing low cost measurement of the application, as describedin FIG. 6 above. However, in further embodiments, this information maybe gathered while performing high cost measurement of the application oranalysis of the high cost measurement, as described in FIGS. 6-8 above.

In some embodiments, the performance information may include CPU loadand GPU load, among other information. The CPU load may be measured interms of overall CPU load, CPU load while performing graphics operations(e.g., CPU load of the graphics framework and/or graphics driver),and/or CPU load while performing non-graphics operations (e.g., overallCPU load minus the graphics CPU load). The amount of CPU time spentwaiting for the GPU or performing graphics related processing may bealso measured. Further, the load of the GPU may be measured.

By performing these measurements, the performance cost of an applicationmay be gauged with respect to CPU and GPU. This may also be extended tomeasure the CPU and GPU cost of specific graphics commands (e.g., usingthe method of FIG. 8). For example, a particular graphics command mayprovide a vertex array in a format that is not accepted by the GPU. TheCPU may accordingly have to convert the vertex array from the originalformat to the GPU accepted format (a graphics related process) andprovide that converted vertex array to the GPU to perform a drawoperation. The performance information may include, for example, theduration, start time, and end time of that graphics related operation,and/or a sum of those durations for all graphics related operations(e.g., by the CPU). Thus, the original graphics command will have aperformance cost in terms of both CPU and GPU, and the overallapplication performance may be bound by the CPU's conversion, the GPU'sdraw operation, or neither, depending on the application. Other CPUgraphics related processing may include texture upload, statevalidation, etc., and similar data may be gathered for that processing.

However, the CPU is generally also performing non-graphics relatedprocessing, e.g., for the application executing on the targetapplication. Where this processing is particularly intensive, it may belimiting the graphics performance of the application.

Thus, the performance data in 904 may be gathered to determine CPU andGPU costs during execution of an application on a target device.

Accordingly, in 906, based on the performance information, it may bedetermined whether the CPU is limiting graphics performance of theapplication. More specifically, it may be determined whether the CPU islimiting graphics performance due to graphics related processing ornon-graphics related processing. Where the CPU is limiting graphicsperformance due to non-graphics related processing, further graphicsanalysis may no longer be required. In some embodiments, if possible,the method may determine which non-graphics related process is limitinggraphics performance (e.g., whether it is execution of the application,a service executed by the target device, another application, etc.).

However, if it is not based on non-graphics related processing (i.e., itis related to graphics processing), it may be determined if the CPU'sgraphics related processing (e.g., associated with the application) islimiting graphics performance or if it is the GPU that is limitinggraphics performance. This may be determined using, for example, themethod of FIG. 8.

In 908, an indication may be provided if the CPU is limiting graphicsperformance of the application. More particularly, the indication mayindicate a) if the CPU is limiting graphics performance due tonon-graphics related processing, b) if the CPU is limiting graphicsperformance due to graphics related processing, or c) if the CPU is notlimiting graphics performance (in which case, it is most likely that theGPU is limiting graphics performance). The indication may furtherinclude an identification of the performance bottleneck that is limitinggraphics performance, whether it is CPU or GPU related, etc.

The method of FIG. 9 may be incorporated with the method of FIG. 8. Moreparticularly, this analysis and indication may be performed along as apart of the analysis performed in FIG. 8. For example, where the CPU islimiting graphics performance due to non-graphics processing, the methodmay stop and the analysis of FIG. 8 may not be performed. However, wherethe CPU is not limiting graphics performance due to non-graphicsprocessing, the method of FIG. 8 may be performed, including determiningwhether CPU graphics related processing is causing performance issuesfor the application executing on the target device.

FIGS. 10A-10E—Exemplary GUIs for Analysis of an Application on a TargetDevice

FIGS. 10A-10E are exemplary screen shots of a program for analyzingexecution of an application on a target device.

As shown in FIG. 10A, the user may select an application to beinvestigated. In this particular example, the user (e.g., theapplication developer) may select “iOS device and application”. The usermay then launch the performance analysis program using the launchbutton.

In FIG. 10B, the program may begin collecting evidence. Moreparticularly, at this stage the application may be executed on thetarget device, and the user may provide input to initiate high costmeasurement. Thus, this Figure illustrates an exemplary GUI forinitiating high cost measurement, e.g., according to the method of FIG.6.

In FIG. 10C, the program may perform high cost measurement, e.g., inresponse to user input in FIG. 10B (although automatic triggering isalso envisioned, as discussed above).

In FIG. 10D, the program may begin analysis and replay using the highcost measurement data, e.g., as described in FIG. 8. As shown in thisparticular screen shot, the fragment shaders are currently beinginvestigated.

Finally, in FIG. 10E, after analysis, the program may providesuggestions for improving graphics performance of the applicationexecuting on the target device. In this particular example, the programsuggests compressing textures to consume less texture memory andbandwidth. The program also suggests simplifying complex vertex shadersto consume fewer execution cycles. Further, the program suggests cullingout vertices that are outside the field of view to avoid wasted GPUeffort. The program also provides a graph of potential increasedperformance after addressing these issues, estimating an increase of 16frames per second (FPS) for uncompressed textures, 12.5 FPS forsimplifying the vertex shader, and 11.5 FPS for culling out of viewverticies. The user may then select the “report” button to export aversion of this data, which may include further suggestions, performancedata, more details, etc.

Thus, FIGS. 10A-10E illustrate exemplary screen shots of a program thatanalyzes graphics performance of an application executing on a targetdevice.

Further Embodiments

In further embodiments, instead of testing for performance issues orbottlenecks for an application, the command stream and playback may beused to test graphics frameworks or graphics drivers. More particularly,they may be used to compare performance of a new version of the graphicsframework or driver against an older version of the graphics frameworkor driver. For example, a same command stream may be executed accordingto a first version of the graphics framework and/or driver and thencompared to execution according to a second version of the graphicsframework and/or driver. Modifications to the command stream may also beused to identify particular differences in performance of the graphicsframework and/or driver. Thus, the both general, pipeline, or peroperation performance differences in new versions (or modified versions)of the graphics framework and/or driver may be identified using themethods described above.

Additionally, while the above is discussed with respect to graphicspipelines or graphics operations, it may be modified to apply to anytype of pipeline or application execution. For example, similar methodsmay be applied to any computation API or command based applications,such as OpenCL or others.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

1.-20. (canceled)
 21. A method comprising: receiving a request tomeasure performance of an application executing on a target device; inresponse to the request, intercepting a command submitted by theapplication to a destination; storing the command in a first buffer;storing a data item referenced by the command in a second buffer;providing the command to the destination; and providing one or morecommands stored in the first buffer, including the intercepted command,and one or more data items stored in the second buffer, including thedata item referenced by the intercepted command, to a host deviceconfigured to measure the performance of the application, the hostdevice being different from the destination, wherein the method isperformed by a computer system including one or more computers.
 22. Themethod of claim 21, wherein: the application is a graphics applicationthat issues one or more graphics commands, the command includes at leastone of the one or more graphics commands, the destination is a graphicssystem including a graphics processing unit (GPU), the data itemincludes graphics data, and the host device is a user device configuredto present results of measuring the performance.
 23. The method of claim22, wherein intercepting the command is performed by an interposelibrary that is injected into the application, the interpose libraryconfigured to intercept the one or more graphics commands between theapplication and the graphics system.
 24. The method of claim 22, whereinthe command is intercepted between a graphics driver of the graphicssystem and the GPU.
 25. The method of claim 22, comprising storingadditional information of the graphics command in the first buffer, theadditional information including a state of the graphics system derivedfrom the graphics command.
 26. The method of claim 22, comprisingstoring additional information of the graphics command in the firstbuffer, the additional information including a flag indicating anexecution parameter of the graphics command.
 27. The method of claim 22,wherein: the graphics data is stored in memory at a memory address andis referenced by the graphics command via a pointer pointing to thememory address, and the method comprises modifying the pointer uponstoring the graphics command in the first buffer and storing thegraphics data in the second buffer, the modifying resulting in thepointer pointing to a location of the graphics data in the secondbuffer.
 28. The method of claim 21, wherein the request indicates thatmeasurement of the performance is a measurement designated as a highcost measurement, the high cost measurement comprising a plurality ofexperiments, wherein in each experiment, the host device modifies arespective parameter of the command.
 29. The method of claim 21, whereinproviding the one or more command and one or more data items to the hostdevice is triggered by detecting that the first buffer is full ordetecting that the second buffer is full.
 30. A system comprising: oneor more computers; and a non-transitory computer-readable medium storinginstructions operable to cause the one or more computers to performoperations comprising: receiving a request to measure performance of anapplication executing on a target device; in response to the request,intercepting a command submitted by the application to a destination;storing the command in a first buffer; storing a data item referenced bythe command in a second buffer; providing the command to thedestination; and providing one or more commands stored in the firstbuffer, including the intercepted command, and one or more data itemsstored in the second buffer, including the data item referenced by theintercepted command, to a host device for measuring the performance ofthe application, the host device being different from the destination.31. The system of claim 30, wherein: the application is a graphicsapplication that issues one or more graphics commands, the commandincludes at least one of the one or more graphics commands, thedestination is graphics system including a graphics processing unit(GPU), the data item includes graphics data, and the host device is auser device configured to present results of measuring the performance.32. The system of claim 31, wherein intercepting the command isperformed by an interpose library that is injected into the application,the interpose library configured to intercept the one or more graphicscommands between the application and the graphics system.
 33. The systemof claim 31, wherein the command is intercepted between a graphicsdriver of the graphics system and the GPU.
 34. The system of claim 31,wherein: the graphics data is stored in memory at a memory address andis referenced by the graphics command via a pointer pointing to thememory address, and the operations comprise modifying the pointer uponstoring the graphics command in the first buffer and storing thegraphics data in the second buffer, the modifying resulting in thepointer pointing to a location of the graphics data in the secondbuffer.
 35. The system of claim 30, wherein providing the one or morecommand and one or more data items to the host device is triggered bydetecting that the first buffer is full or detecting that the secondbuffer is full.
 36. A non-transitory computer-readable medium storinginstructions operable to cause one or more computers to performoperations comprising: receiving a request to measure performance of anapplication executing on a target device; in response to the request,intercepting a command submitted by the application to a destination;storing the command in a first buffer; storing a data item referenced bythe command in a second buffer; providing the command to thedestination; and providing one or more commands stored in the firstbuffer, including the intercepted command, and one or more data itemsstored in the second buffer, including the data item referenced by theintercepted command, to a host device for measuring the performance ofthe application, the host device being different from the destination.37. The non-transitory computer-readable medium of claim 36, wherein:the application is a graphics application that issues one or moregraphics commands, the command includes at least one of the one or moregraphics commands, the destination is graphics system including agraphics processing unit (GPU), the data item includes graphics data,and the host device is a user device configured to present results ofmeasuring the performance.
 38. The non-transitory computer-readablemedium of claim 37, wherein intercepting the command is performed by aninterpose library that is injected into the application, the interposelibrary configured to intercept the one or more graphics commandsbetween the application and the graphics system.
 39. The non-transitorycomputer-readable medium of claim 37, wherein the command is interceptedbetween a graphics driver of the graphics system and the GPU.
 40. Thenon-transitory computer-readable medium of claim 37, wherein: thegraphics data is stored in memory at a memory address and is referencedby the graphics command via a pointer pointing to the memory address,and the operations comprise modifying the pointer upon storing thegraphics command in the first buffer and storing the graphics data inthe second buffer, the modifying resulting in the pointer pointing to alocation of the graphics data in the second buffer.